{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview of actor critic method\n",
    "\n",
    "The actor critic method is one of the most popular algorithms in deep reinforcement learning. Several modern deep reinforcement learning algorithms are designed based on the actor critic methods. The actor critic method lies in the intersection of value based and policy based methods. That is, it takes advantage of both value based and policy based methods.\n",
    "\n",
    "In this section, without getting into more details, first, let's get a basic understanding of how actor critic method works and in the next section, we will get into more details and understand the math behind the actor critic methods. \n",
    "\n",
    "Actor critic, as the name suggests consists of two types of networks called actor network and critic network. The role of the actor network is to find an optimal policy while the role of the critic network is to evaluate the policy produced by the actor network. So, we can think of, critic network as a form of feedback network which evaluates and guides the actor network in finding the optimal policy as shown below:\n",
    "\n",
    "![title](Images/1.png)\n",
    "\n",
    "Okay, what's really actor and critic network? how it works together and improve the policy? The actor network is basically the policy network and it finds the optimal policy using a policy gradient method. The critic network is basically the value network and it estimates the state value. Thus using its state value, the critic network evaluates the action produced by actor network and sends its feedback to the actor. Based on the critic's feedback, actor network updates its parameter.\n",
    "\n",
    "Thus, in the actor critic method, we use two networks - actor network (policy network) which computes the policy and the critic network (value network) which evaluates the policy produced by actor network by computing the value function (state values). Isn't this similar to something we just learned in the previous chapter?\n",
    "\n",
    "Yes! If you could recollect it is similar to the policy gradient method with the baseline (reinforce with baseline) we learned in the previous chapter. Similar to reinforce with baseline, here also we have an actor (policy network) and a critic (value network) network. However, actor critic is NOT exactly similar to reinforce with baseline. In the reinforce with baseline method, we learned that we use value network as the baseline and it helps to reduce the variance in the gradient updates. In the actor-critic method as well, we use the critic to reduce variance in the gradient updates of the actor but also critic helps to improve the policy iteratively in an online fashion. The distinction between these two will be made clear in the next section.\n",
    "\n",
    "Now that we have a basic understanding of what is actor critic method, in the next section we will learn how exactly the actor critic method works in detail. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
